Image processing system, image processing method, and computer readable medium

ABSTRACT

There is provided an image processing system configured to correct an image of an object inside a physical body. The image processing system includes an object image obtaining section that obtains an object image formed by light from the object, a depth identifying section that identifies a depth from a surface of the physical body to the object, a distance information identifying section that identifies distance information indicating a distance from an image capturing section capturing the object image to the surface of the physical body, and an image correcting section that corrects the object image according to the distance information and the depth.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims priority from a Japanese PatentApplication No. 2007-314496 filed on Dec. 5, 2007, the contents of whichare incorporated herein by reference.

BACKGROUND

1. Technical Field

The present invention relates to an image processing system, an imageprocessing method and a computer readable medium. More particularly, thepresent invention relates to an image processing system and an imageprocessing method for correcting an image of an object inside a physicalbody and to a computer readable medium storing therein a program for usewith the image processing system.

2. Related Art

An endoscope disclosed in Japanese Patent Application Publication No.10-165365 processes an image signal representing a predetermined rangeby using a point spread function of an objective optical system in orderto clarify a blur of the image. Such a process is expected to be capableof clarifying a blur of the image caused by an insufficient focal depthof the objective optical system while increasing the amount of the lightincident on the image capturing section.

A known measuring apparatus includes: a CCD camera that obtains binaryinformation of a target physical body, to which artificial sunshine isirradiated in a darkroom, and displays the target physical body usingdots on a CRT display; and a pair of laser emitters that are displayedon the target physical body as reference pointers as disclosed in, forexample, Japanese Patent Application Publication No. 7-250538. Themeasuring apparatus calculates the area and peripheral length of thetarget physical body displayed on the CRT display with reference to theratio between the distance between the reference pointers P1 and P2displayed by using dots on the CRT display and the distance between thereference pointers on the actual target physical body, and then performsa comparison operation on the calculated area and peripheral length byusing the ratio in order to know the position, shape and size of thetarget physical body. Furthermore, a known distance measuring apparatusincludes: a CCD camera that is constituted by a fixed focus lens and aCCD of a photoelectric converter and that has a horizontal field angleof α and a vertical field angle of β; and laser light sources that areprovided on the respective sides of the CCD camera with respect to anoptical axis X-X with a predetermined interval LA/2 therebetween andemit a set of radial vertical parallel leaser light rays that have aradiation angle of θ, as disclosed in, for example, Japanese PatentApplication Publication No. 2000-230807.

For example, when an object such as a blood vessel inside a livingorganism is observed with the use of light from the object, only ablurry image is produced for the object. Specifically speaking, thelight from the object is scattered while traveling through the livingorganism and the image of the object is vaguely outlined. Here, whenmaking a diagnosis or performing an operation with the help of enendoscope, a medical doctor desires to accurately know the position ofan object such as a blood vessel. Therefore, it is demanded to correctthe blurry outline of the image of the object and thus provide themedical doctors with a clear image. The techniques disclosed in theabove-mentioned three publications, however, cannot correct blurryimages of objects inside physical bodies.

SUMMARY

Therefore, it is an object of an aspect of the innovations herein toprovide an image processing system, an image processing method, and acomputer readable medium which are capable of overcoming the abovedrawbacks accompanying the related art. The above and other objects canbe achieved by combinations described in the independent claims. Thedependent claims define further advantageous and exemplary combinationsof the innovations herein.

To solve the above-mentioned problem, according to the first aspectrelated to the innovations herein, one exemplary image processing systemmay include an object image obtaining section that obtains an objectimage formed by light from an object inside a physical body, a depthidentifying section that identifies a depth from a surface of thephysical body to the object, a distance information identifying sectionthat identifies distance information indicating a distance from an imagecapturing section capturing the object image to the surface of thephysical body, and an object image correcting section that corrects theobject image according to the distance information and the depth.

According to the second aspect related to the innovations herein, oneexemplary image processing method may include obtaining an object imageformed by light from an object inside a physical body, identifying adepth from a surface of the physical body to the object, identifyingdistance information indicating a distance from an image capturingsection capturing the object image to the surface of the physical body,and correcting the object image according to the distance informationand the depth.

According to the third aspect related to the innovations herein, oneexemplary computer readable medium stores therein a program for use withan image processing system. When executed, the program causes the imageprocessing system to function as an object image obtaining section thatobtains an object image formed by light from an object inside a physicalbody, a depth identifying section that identifies a depth from a surfaceof the physical body to the object, a distance information identifyingsection that identifies distance information indicating a distance froman image capturing section capturing the object image to the surface ofthe physical body, and an image correcting section that corrects theobject image according to the distance information and the depth.

The summary clause does not necessarily describe all necessary featuresof the embodiments of the present invention. The present invention mayalso be a sub-combination of the features described above. The above andother features and advantages of the present invention will become moreapparent from the following description of the embodiments taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of an image processing system 10 relatingto an embodiment of the present invention, together with a specimen 20.

FIG. 2 illustrates an exemplary block configuration of an imageprocessing section 140.

FIG. 3 schematically illustrates how light is, for example, reflectedinside the specimen 20.

FIG. 4 schematically illustrates how light from the inside of thespecimen 20 becomes blurry.

FIG. 5 illustrates exemplary blood vessel images 560 and 570 obtained asa result of correction by an object image correcting section 220.

FIG. 6 illustrates an exemplary configuration of an image capturingsection 110.

FIG. 7 illustrates exemplary spectral sensitivity characteristics offirst, second and third light receiving elements 851, 852 and 853.

FIG. 8 illustrates an exemplary configuration of a light irradiatingsection 150.

FIG. 9 illustrates an exemplary configuration of a source-side filtersection 1020.

FIG. 10 illustrates the image capturing timings of the image capturingsection 110 and exemplary images generated by the image processingsection 140.

FIG. 11 illustrates how to generate a motion-compensated surface image.

FIG. 12 illustrates another exemplary method to generate amotion-compensated surface image.

FIG. 13 illustrates an exemplary hardware configuration of the imageprocessing system 10 relating to the embodiment of the presentinvention.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Some aspects of the invention will now be described based on theembodiments, which do not intend to limit the scope of the presentinvention, but exemplify the invention. All of the features and thecombinations thereof described in the embodiment are not necessarilyessential to the invention.

FIG. 1 illustrates an example of an image processing system 10 relatingto an embodiment of the present invention, together with a specimen 20.The image processing system 10 is configured to correct an image of anobject inside a physical body, according to the depth of the object. Theimage processing system 10 includes an endoscope 100, an imageprocessing section 140, an output section 180, a control section 105, alight irradiating section 150, and an ICG injecting section 190. In FIG.1, the reference character A indicates an enlargement view of an endportion 102 of the endoscope 100. The control section 105 includes animage capturing control section 160 and a light irradiation controlsection 170.

The ICG injecting section 190 injects indocyanine green (ICG) into thespecimen 20, where indocyanine green serves as an luminescence substanceand the specimen 20 is shown as an example of the physical body in thepresent invention. In the present embodiment, ICG is introduced as anexample of the luminescence substance. Nonetheless, other fluorescentsubstances than ICG may be used as the luminescence substance. ICG isexcited, for example, by an infra-red ray having a wavelength of 750 nm,to emit fluorescence with a broad spectrum having as its middle awavelength of 810 nm.

When the specimen 20 is a living organism, the ICG injecting section 190intravenously injects ICG into the blood vessels of the living organism.The image processing system 10 captures images of the blood vessels ofthe living organism with the use of luminescence light emitted from theICG. Here, the luminescence light is shown as an example of the lightfrom the physical body and includes fluorescence and phosphorescence.The luminescence light includes luminescence light produced bychemiluminescence, triboluminescence, or thermoluminescence in additionto photoluminescence caused by light such as excitation light. Here, ablood vessel may be shown as an example of the object in the presentinvention.

The ICG injecting section 190 injects ICG into the specimen 20, underthe control of, for example, the control section 105, in such a mannerthat the ICG concentration within the living organism remainssubstantially constant. The specimen 20 may be a living organism, forexample. In the specimen 20, there is an object such as a blood vessel.The image processing system 10 relating to the present embodimentdetects the depth of an object from the surface of the specimen 20,where the object is underneath the surface of the specimen 20 (includingthe internal surface of an organ or the like). In addition, the imageprocessing system 10 corrects the blurry image of the object accordingto the detected depth.

The endoscope 100 includes an image capturing section 110, a light guide120, and a forceps entrance 130. The end portion 102 of the endoscope100 has an objective lens 112, which forms a portion of the imagecapturing section 110. The end portion 102 also has a light exit 124 anda light exit 126, which form a portion of the light guide 120. The endportion 102 of the endoscope 100 further has a nozzle 138.

A forceps 135 is inserted into the forceps entrance 130, so that theforceps entrance 130 guides the forceps 135 through the end portion 102.The forceps 135 may have an end with any of various shapes. In additionto forcipes, a variety of treatment tools for treating living organismsmay be inserted into the forceps entrance 130. The nozzle 138 sends outwater or air.

The light irradiating section 150 generates light to be irradiatedthrough the end portion 102 of the endoscope 100. The light generated bythe light irradiating section 150 contains an infra-red ray andirradiation light, where the infra-red ray is shown as an example ofexcitation light that excites the luminescence substance in the specimen20 to emit the luminescence light and the irradiation light is lightirradiated to the specimen 20. The irradiation light includesR-component light, G-component light and B-component light, for example.

The image capturing section 110 captures images by using theluminescence light emitted from the luminescence substance andreflection light which is a portion of the irradiation light reflectedby the object. The image capturing section 110 may include atwo-dimensional image capturing device such as a CCD and an opticalsystem, where the optical system contains the objective lens 112. Whenthe luminescence substance emits infrared light, the image capturingsection 110 can capture infrared light images. When the object isirradiated with light containing all of R, G and B components, forexample, white light, the image capturing section 110 can capturevisible light images.

The light from the object can be exemplified by luminescence light suchas fluorescence or phosphorescence emitted by the luminescence substancein the object, or one of reflection light which is a reflected portionof the light irradiated to the object and transmission light which is atransmitted portion of the light irradiated to the object. In otherwords, the image capturing section 110 captures images of the object byusing the light emitted from the luminescence substance in the object,the light reflected by the object, or the light transmitted by theobject. Here, the image capturing section 110 can separately receive, ina time-or space-sharing manner, component light in the R wavelengthrange, component light in the G wavelength range, component light in theB wavelength range, and light in the luminescence light wavelengthrange.

The image capturing section 110 can capture images of the object byusing any one of various methods, in addition to the method using thelight from the object. For example, the image capturing section 110 maycapture images of the object with the use of electromagnetic radiationsuch as X-rays or γ rays, or radiation containing corpuscular rays suchas alpha rays. Alternatively, the image capturing section 110 maycapture images of the object by using electromagnetic waves, radio wavesor sound waves having a variety of wavelengths.

The light guide 120 can be formed by an optical fiber, for example. Thelight guide 120 is designed to guide the light generated by the lightirradiating section 150 to the end portion 102 of the endoscope 100. Thelight guide 120 may include the light exits 124 and 126 formed in theend portion 102. The light generated by the light irradiating section150 is irradiated to the specimen 20 through the light exits 124 and126.

The light from the light exit 124 is irradiated so as to cover the imagecapturing range of the image capturing section 110. For example, thelight from the light exit 124 is at least irradiated to a range, asdefined by a line 114, from which the image capturing section 110 canreceive light. On the other hand, the light from the light exit 126 isirradiated to the surface of a physical body in a directionsubstantially parallel to the image capturing direction of the imagecapturing section 110. For example, the light from the light exit 126may be spot light, as indicated by a line 128, irradiated within theimage capturing range of the image capturing section 110. The spot lightmainly includes light substantially parallel to the image capturingdirection of the image capturing section 110. With the use of the lightfrom the light exit 124 and the light from the light exit 126, the imagecapturing section 110 can at least capture images of the surface of thephysical body.

The image processing section 140 processes image data obtained from theimage capturing section 110. The output section 180 outputs the imagedata processed by the image processing section 140. The image capturingcontrol section 160 controls the image capturing operation by the imagecapturing section 110. The light irradiation control section 170controls the light irradiating section 150, under the control of theimage capturing control section 160. For example, when the imagecapturing section 110 captures images by using an infrared ray and theirradiation light in a time-sharing manner, the light irradiationcontrol section 170 controls the light irradiating section 150 such thatthe irradiating timings of the infrared ray and irradiation light aresynchronized with the image capturing timings by the image capturingsection 110.

The image processing section 140 detects the depth of the object fromthe surface and the distance to the surface of the physical body fromthe image capturing section 110, with reference to the images capturedby the image capturing section 110. According to the detected depth anddistance, the image processing section 140 corrects the images of theobject captured by the image capturing section 110.

FIG. 2 illustrates an exemplary block configuration of the imageprocessing section 140. The image processing section 140 includes anobject image obtaining section 210, a light image obtaining section 212,a surface image obtaining section 214, an object region identifyingsection 216, an object image correcting section 220, a correction table222, a distance information identifying section 240, a motionidentifying section 270, a subject image generating section 280, adisplay control section 260, and a depth identifying section 230. Theobject image correcting section 220 includes a correction valuetransforming section 224 and an object image correction section 226.

The light image obtaining section 212 obtains a light image, which is animage formed by the light from the object such as a blood vessel insidethe specimen 20. Specifically speaking, the light image obtainingsection 212 obtains a light image, which is an image formed by the lightfrom the object inside a physical body. To be more specific, the lightimage obtaining section 212 obtains, as a light image, the imagecaptured by the image capturing section 110 by using the light from theobject.

When the light from the object is the luminescence light emitted by theluminescence substance, the light image obtained by the light imageobtaining section 212 includes an image of an object present between thesurface of the physical body and a depth that can be reached byexcitation light used to excite the luminescence substance. For example,the excitation light for the luminescence substance irradiated from theend portion 102 of the endoscope 100 has a wavelength of 750 nm.Therefore, this excitation light reaches a relatively large depth in thespecimen 20 (for example, a depth of approximately several centimeters).In this case, the light image obtained by the light image obtainingsection 212 includes an image of a blood vessel present at a relativelylarge depth in the specimen 20. Note that a blood vessel image may be anexample of the light image in the present invention.

The excitation light excites a luminescence substance present within thedepth that can be reached by the excitation light. Therefore, the lightimage obtained by the light image obtaining section 212 includes animage of a blood vessel present within the depth that can be reached bythe excitation light. Here, the deeper a blood vessel is positioned, themore significantly the fluorescence from the blood vessel is scatteredby the specimen 20. Hence, the deeper a blood vessel is positioned, themore blurry the image of the blood vessel is.

When the light from the object is the reflection light from the object,the light image obtained by the light image obtaining section 212includes an object present within a depth that is reached by theirradiation light to the object and at which the irradiation light isreflected. Here, the depth that can be reached by the irradiation lightto the physical body is dependent on the wavelength of the irradiationlight. Specifically speaking, red light can reach a larger depth in aphysical body than blue light, green light can reach an intermediatedepth, and infrared light can reach a larger depth in a physical bodythan red light. Hence, the light image includes an image of an objectpresent between the surface of a physical body and a reachable depth,which is dependent on the wavelength range of light irradiated to thephysical body.

As described above, the light image obtaining section 212 obtains aplurality of light images each of which is captured by using light, froman object such as a blood vessel, in one of a plurality of differentwavelength ranges. These wavelength ranges can be defined as needed andcan be exemplified by a red range whose middle wavelength corresponds tothe R component of visible light, a green range whose middle wavelengthcorresponds to the G component of visible light, and a blue range whosemiddle wavelength corresponds to the B component of visible light.Alternatively, these wavelength ranges may be obtained by dividing thewavelength range of the fluorescence from the ICG into a plurality ofwavelength ranges and can be exemplified by a long wavelength range, anintermediate wavelength range, and a short wavelength range included inthe wavelength range of the fluorescence from the ICG.

The depth identifying section 230 identifies the depth of an object suchas a blood vessel, with reference to what is shown by a plurality oflight images. For example, the depth identifying section 230 detects thedepth of an object such as a blood vessel by making use of opticalcharacteristics that light of a different wavelength can reach adifferent depth in the specimen 20, or by making use of differentoptical characteristics that light of a different wavelength exhibits adifferent absorptance in the specimen 20. Specifically speaking, thedepth identifying section 230 judges that a blood vessel that can beobserved in a blue-range light image is present within a depth at whichlight having a wavelength in the blue range is reflected. Similarly, thedepth identifying section 230 judges that a blood vessel that can beobserved in a green-range light image is present within a depth at whichlight having a wavelength in the green range is reflected. Furthermore,the depth identifying section 230 judges that a blood vessel that can beobserved in a red-range light image is present within a depth at whichlight having a wavelength in the red range is reflected.

Referring to the fluorescence emitted from the ICG in the blood vessel,light in the long wavelength range is absorbed less than light in theshort wavelength range. Therefore, the depth identifying section 230estimates the depth of the blood vessel with reference to the ratio inbrightness between the blood vessel images included in the light imagesrespectively formed by the light in the long, intermediate and shortwavelength ranges. For example, when a blood vessel image included in alight image formed by the light in the short wavelength range is dark inrelation to the brightness of a blood vessel image included in a lightimage formed by the light in the long wavelength range, the depthidentifying section 230 determines that the blood vessel is present at adeep position. Conversely, when a blood vessel image included in a lightimage formed by the light in the short wavelength range is bright inrelation to the brightness of a blood vessel image included in a lightimage formed by the light in the long wavelength range, the depthidentifying section 230 estimates that the optical path that wouldabsorb the light in the short wavelength range is not long and that theblood vessel is thus present at a shallow position.

As described above, the depth identifying section 230 can detect thedepth of an object such as a blood vessel by utilizing the opticalcharacteristic that how deep in a physical body light can reach (thedepth at which the light is reflected) is different depending on thewavelength of the light. In this case, the light image obtaining section212 may obtain, as the light image, an image formed by the reflectionlight reflected by the object. The light image obtaining section 212 mayobtain a plurality of light images, each of which is formed by light inone of a plurality of different wavelength ranges in the wavelengthrange of light reflected by the object when white light is irradiated tothe object. Alternatively, the light image obtaining section 212 mayobtain a plurality of light images, each of which is formed by lightreflected by the object when light in one of a plurality of differentwavelength ranges is irradiated to the object.

When the depth identifying section 230 detects the depth of an objectsuch as a blood vessel by using the optical characteristic thatfluorescence emitted at a deep portion of a physical body is absorbed ata different ratio depending on the wavelength as discussed above, thelight image obtaining section 212 obtains a plurality of light images,each of which is formed by light in one of a plurality of differentwavelength ranges included in the wavelength range of the light emittedfrom the luminescence substance in the object. The light irradiated tothe object is emitted by the light irradiating section 150 andirradiated through the light exit 124.

As explained above, since light images have information regarding howdeep light can reach, the depth identifying section 230 can calculatethe depth of an object by comparing or performing an operation on thebrightness (luminance) levels of the objects in the light images. Forexample, the object region identifying section 216 identifies imageregions showing an object in a plurality of light images constitutingeach light image set. The depth identifying section 230 then identifiesthe depth from the surface to the object based on the luminance levelsof the image regions identified by the object region identifying section216.

For example, the depth identifying section 230 identifies the depth byreferring to the ratio of the luminance of the image region in the shortwavelength range light image to the luminance of the image region in thelong wavelength range light image. Alternatively, the depth identifyingsection 230 may identify the depth based on the maximum or averageluminance among the image regions.

As a further alternative example, the depth identifying section 230 mayidentify the depth by referring to the luminance change ratios at theedges of the object image regions. This luminance change ratio can beexpressed as, for example, a derivative of the luminance that variesaccording to the position (distance) in the image space. This derivativeis an example of numerical representation indicating how blurry theobject is in the image region. As the derivative increases, the blurdecreases and the estimated position of the object becomes shallower.

The luminance change ratio can be expressed as, for example, a halfbandwidth of the distribution of the luminance that varies according tothe position (distance) in the image space. As the half bandwidthincreases, the blur increases. As the half bandwidth decreases, theestimated position of the object becomes shallower. In theabove-described manner, the depth identifying section 230 can identifythe depth from the surface of the physical body to the object.

The distance information identifying section 240 identifies the distancefrom the image capturing section 110, which captures light images, tothe surface of a physical body. In the present invention, distanceinformation may be a distance in the real space itself or an indicatorof the distance in the real space. In the following description, areference to “a distance” indicates the distance from the imagecapturing section 110 to the surface of the physical body, unlessotherwise stated.

For example, the surface image obtaining section 214 obtains an imageincluding a surface image, which is an image of the surface of aphysical body formed by light irradiated to the surface of thephysical-body in a direction substantially parallel to the imagecapturing direction of the image capturing section 110. The distanceinformation identifying section 240 then identifies the distance basedon the size of the surface image included in the image obtained by thesurface image obtaining section 214. Here, the substantially parallellight may be the spot light emitted from the light irradiating section150 and irradiated through the light exit 126.

The object image obtaining section 210 obtains an object image capturedby using the light from the object. The image capturing section 110 maycapture an image of the object through the surface. The object imageobtaining section 210 obtains the image of the object which is capturedthrough the surface. For example, the object image obtaining section 210obtains an object image captured by the image capturing section 110 byusing the luminescence light from the object.

The object image correcting section 220 corrects the object imageaccording to the distance identified by the distance informationidentifying section 240 and the depth identified by the depthidentifying section 230. Specifically speaking, the object imagecorrecting section 220 corrects the object image according to thedistance identified by the distance information identifying section 240and the depth identified by the depth identifying section 230. Morespecifically, the object image correcting section 220 corrects thespread of the object image according to the distance identified by thedistance information identifying section 240 and the depth identified bythe depth identifying section 230. To be further specific, the objectimage correcting section 220 corrects the spread of the object image,which is caused because the light from the object is scattered betweenthe object and the surface, according to the distance identified by thedistance information identifying section 240 and the depth identified bythe depth identifying section 230.

The corrected image obtained as a result of the correction by the objectimage correcting section 220 is supplied to the output section 180 anddisplayed by the output section 180. The correction table 222 stores acorrection value in association with a depth from a surface to anobject, where the correction value is used to correct spread of anobject image. The object image correcting section 220 corrects thespread of the object image, with reference to the distance identified bythe distance information identifying section 240, the depth identifiedby the depth identifying section 230, and a corresponding correctionvalue stored on the correction table 222.

The surface image obtaining section 214 obtains an image of a physicalbody surface that is captured by the image capturing section 110, as oneexample. In other words, the surface image obtaining section 214 obtainsthe same image as visually observed. For example, the surface imageobtaining section 214 obtains, as the surface image, an image capturedby the image capturing section 110 by using a portion of the irradiationlight which is reflected by the surface of the physical body. Forexample, the surface image obtaining section 214 obtains, as the surfaceimage, the image captured by the image capturing section 110 by using aportion of the irradiation light which is reflected by the surface ofthe physical body, where the irradiation light is white light.

The image capturing section 110 may capture an object image and asurface image at different timings. For example, the image capturingsection 110 may successively capture a surface image with visible lightby irradiating white light and capture an object image by irradiatingexcitation light in place of white light at a predetermined timing. Inthis case, the motion identifying section 270 identifies a motion of theobject made between the excitation light irradiation timing and thewhite light irradiation timing. The subject image generating section 280generates a surface image which is supposed to be obtained at theexcitation light irradiation timing, based on the surface image obtainedby the irradiation of the white light and the motion identified by themotion identifying section 270. How the control section 105, the imagecapturing section 110, the light irradiating section 150, the motionidentifying section 270 and the subject image generating section 280function and operate to capture an object image and a surface image in atime-sharing manner is described in more detail with reference to FIG. 6and subsequent drawings.

The display control section 260 controls how a surface image and acorrected image are displayed through the output section 180. Forexample, the display control section 260 controls how to display theobject image corrected by the object image correcting section 220,according to the depth identified by the depth identifying section 230.Specifically speaking, the display control section 260 changes thebrightness or color of the object image corrected by the object imagecorrecting section 220 according to the depth and causes the outputsection 180 to display the resultant image. Alternatively, the displaycontrol section 260 may cause the output section 180 to display asurface image and a corrected image next to each other. Furthermore, thedisplay control section 260 may cause the output section 180 to displaya numerical value indicating object depth information.

The following describes the operations of the constituents of the objectimage correcting section 220. The object image correcting section 220includes the correction value transforming section 224 and the objectimage correction section 226. The correction table 222 stores acorrection value for the real space in association with a depth of anobject from a surface. In other words, the correction table 222 stores acorrection value for the real scale. In this case, the correction valuetransforming section 224 transforms a correction value for the realspace stored on the correction table 222 into a correction valueappropriate for an object image, according to the distance identified bythe distance information identifying section 240.

The object image correction section 226 corrects the spread of theobject image by using the correction value appropriate for the objectimage obtained by the correction value transforming section 224. In thismanner, the object image correcting section 220 can appropriatelycorrect the blurry object image according to the position of thesurface.

The depth identifying section 230 may identify the depth of each of aplurality of objects from the surface. The depth identifying section 230may calculate the depth of each of a plurality of objects from thesurface. The object image correcting section 220 may correct spread ofan image of each of a plurality of objects in an object image, accordingto the depth of each of the plurality of objects.

FIG. 3 schematically illustrates how light is, for example, reflectedinside the specimen 20. Inside the specimen 20, there is a blood vessel810, which is an exemplary object. Here, ICG, which is an exemplaryluminescence substance, is injected into a blood vessel. Therefore, ICGis in the blood vessel 810. The specimen 20 is irradiated with infraredlight Iir, which is designed to excite the ICG, and red light Ir, greenlight Ig, and blue light Ib which are irradiated to the blood vessel810, which is an exemplary object.

The infrared light Iir can reach a relatively deep position (having adepth dir) in the specimen 20, and excites the ICG in the blood vessel810 from the surface to the depth dir. Therefore, an image of the bloodvessel 810 within the depth dir is captured with the use of fluorescenceRf emitted from the ICG, to produce an object image. Note that the imageof the blood vessel 810, which is obtained as an object image, isblurry.

The red light Ir reaches a depth dr and is reflected in the vicinity ofthe depth dr. Therefore, red reflection light Rr of the red light Ircontains image information regarding a portion of the blood vessel 810in the vicinity of the depth dr. The image of the blood vessel 810formed by the red reflection light Rr is obtained as a light imageformed by the red wavelength range light. This light image includes animage showing the portion of the blood vessel 810 in the vicinity of thedepth dr.

The green light Ig reaches a depth dg and is reflected in the vicinityof the depth dg. Therefore, green reflection light Rg of the green lightIg contains image information regarding a portion of the blood vessel810 in the vicinity of the depth dg. The image of the blood vessel 810formed by the green reflection light Rg is obtained as a light imageformed by the green wavelength range light. This light image includes animage showing the portion of the blood vessel 810 in the vicinity of thedepth dg.

The blue light Ib reaches a depth db and is reflected in the vicinity ofthe depth db. Therefore, blue reflection light Rb of the blue light Ibcontains image information regarding a portion of the blood vessel 810in the vicinity of the depth db. The image of the blood vessel 810formed by the blue reflection light Rb is obtained as a light imageformed by the blue wavelength range light. This light image includes animage showing the portion of the blood vessel 810 in the vicinity of thedepth db.

As mentioned above, the depth identifying section 230 can identify thedepth of the blood vessel 810 based on the light images formed by thered reflection light Rr, the green reflection light Rg, and the bluereflection light Rb. The object image correcting section 220 can correctthe object image formed by the fluorescence Rf according to the depthidentified by the depth identifying section 230.

FIG. 4 schematically illustrates how the light from the inside of thespecimen 20 becomes blurry. It is assumed that a point light source ispresent at a position 410 in the specimen 20 away from the surface 400of the specimen 20 by a depth d1. The light from the point light sourceto the image capturing section 110 is scattered by the specimen 20 whiletraveling from the position 410 to the surface 400, so as to have aluminance distribution designated by h1 at the surface 400. Here, xdenotes a real-space x coordinate within a plane vertical to the imagecapturing direction.

Here, it is also assumed that a point light source is present at aposition 420 in the specimen 20 away from the surface 400 of thespecimen 20 by a depth d2. The light from the point light source to theimage capturing section 110 is scattered by the specimen 20 whiletraveling from the position 420 to the surface 400, so as to have aluminance distribution designated by h2 at the surface 400. In FIG. 4,the spread is shown one-dimensionally, but the luminance distribution atthe surface is two-dimensional.

As seen from the distributions in FIG. 4, the deeper the point lightsource is positioned, the broader the light from the point light sourceis at the surface. For this reason, an object image formed by light froman object at the position 420 is more blurry than an object image formedby light from an object at the position 410. Here, since the spread isdependent the depth from the surface to the point light source, thedegree of the spread is defined in the real space scale. Therefore, apoint spread function in the real space is associated with a depth of anobject. Consequently, when the depth identifying section 230 identifiesa depth in terms of the image capturing direction of the image capturingsection 110 and the distance information identifying section 240identifies a distance to a surface, the object image correcting section220 can correct spread of an object image, with reference to a pointspread function associated with the identified depth and distance.

For example, the correction table 222 stores an inverse filter of apoint spread function whose parameter is a depth. The correction valuetransforming section 224 transforms the stored inverse filter into aninverse filter that is applied to an image, according to a distanceidentified by the distance information identifying section 240. Theobject image correction section 226 corrects each object imageidentified by the object region identifying section 216 by using aninverse filter obtained by the correction value transforming section224.

Alternatively, the correction table 222 may store an inverse filter of apoint spread function whose parameters are a depth and a distance. Theobject image correction section 226 may correct each object imageidentified by the object region identifying section 216 by using aninverse filter associated with the distance identified by the distanceinformation identifying section 240 and the depth identified by thedepth identifying section 230.

FIG. 5 illustrates exemplary blood vessel images 560 and 570 obtained asa result of the correction by the object image correcting section 220.An image 500 captured by the image capturing section 110 includes blurryblood vessel images 510 and 520. Note that the blood vessel image 510shows a blood vessel at a shallower position than the blood vessel image520 and that the blood vessel image 510 is less blurry than the bloodvessel image 520.

The image obtained by the surface image obtaining section 214 includes asurface image formed by the spot light irradiated in a directionsubstantially parallel to the image capturing direction of the imagecapturing section 110. The surface image captured by the image capturingsection 110 with the use of a portion of the spot light which isreflected by the surface shows a portion on the surface to which thespot light is irradiated. The distance information identifying section240 identifies the distance between the image capturing section 110 andthe surface by referring to the size of the surface image formed by thespot light. For example, the distance information identifying section240 may identify the distance between the image capturing section 110and the surface by referring to at least one of the diameter and thearea of the surface image formed by the spot light.

The spot light emitted through the light exit 126 may have asubstantially circular projected image on a plane perpendicular to itstravel direction. When the shape of the spotlight is known, the distanceinformation identifying section 240 may identify the distance betweenthe image capturing section 110 and the surface further with referenceto the shape of the surface image formed by the spot light. For example,when the spot light has a circular shape, the distance informationidentifying section 240 may identify the distance between the imagecapturing section 110 and the surface with reference to the minor axisof the surface image. Furthermore, when the spot light has a circularshape, the distance information identifying section 240 may identify thedistance between the image capturing section 110 and the surface furtherwith reference to the degree with which the surface image is similar toan ellipse. With such a configuration, the distance informationidentifying section 240 can identify the distance based on the size andshape of the surface image even when the spot light is not verticallyincident on the surface.

The depth identifying section 230 can also calculate a depth from asurface to a blood vessel in the above-described manner. The objectregion identifying section 216 extracts the regions of the blood vesselimages 510 and 520 from the image 500, as blood vessel regions. Theobject image correcting section 220 corrects the regions of the bloodvessel images 510 and 520 identified by the object region identifyingsection 216 with the use of inverse filters associated with the distancefrom the surface to the image capturing section 110 and the depths ofthe blood vessels identified by the depth identifying section 230, asdescribed above.

To sum up, the object image correcting section 220 increases thecorrection to be performed on spread of an object image as a depthincreases. The object image correcting section 220 also increases thecorrection to be performed on spread of an object image as a distanceindicated by distance information decreases. For example, since theblood vessel image 510 shows a blood vessel at a shallower position thanthe blood vessel image 520, the object image correcting section 220corrects the blur of the blood vessel image 520 more significantly thanthe blur of the blood vessel image 510.

The object image correcting section 220 corrects the blurry blood vesselimage 510 and outputs a blood vessel image 560 to the display controlsection 260. In addition, the object image correcting section 220corrects the blurry blood vessel image 520 and outputs a blood vesselimage 570 to the display control section 260. The display controlsection 260 causes the output section 180 to display the depths from thesurface, for example, by using different gradation levels or colors. Useof the image processing system 10 relating to the present embodiment maymake it possible for medical doctors to have a clear view of internalblood vessels, which are not visible from the surface, for example, whenthe medical doctors perform operations and the like with their eyes onthe display provided by the output section 180. In addition, the medicaldoctors may advantageously be capable of performing operations and thelike while being provided with information regarding the depths of theinternal blood vessels.

In the present embodiment, the light irradiation control section 170 maycontrol the light irradiation intensity according to the distanceidentified by the distance information identifying section 240. Forexample, the light irradiation control section 170 may decrease theintensity of the light irradiated from the light irradiating section 150as the distance identified by the distance information identifyingsection 240 decreases. The light irradiation control section 170 maystop the light irradiation from the light irradiating section 150, whenthe distance identified by the distance information identifying section240 exceeds a predetermined value. Here, the distance informationidentifying section 240 may identify the distance to the surface, forexample, based on irradiation of ultrasound wave or laser to thesurface.

FIG. 6 illustrates an exemplary configuration of the image capturingsection 110. The image capturing section 110 includes an objective lens112, an image capturing device 810, a spectral filter section 820, and areception-side excitation light cut filter section 830. The imagecapturing device 810 includes a plurality of first light receivingelements 851 having a first light receiving element 851 a, a pluralityof second light receiving elements 852 having second light receivingelements 852 a and 852 b, and a plurality of third light receivingelements 853 having a third light receiving element 853 a.

The following describes the functions and operations of the constituentsof the image capturing section 110. To make the following descriptionsimple, the plurality of first light receiving elements 851 may becollectively referred to as the first light receiving elements 851, theplurality of second light receiving elements 852 may be collectivelyreferred to as the second light receiving elements 852, and theplurality of third light receiving elements 853 may be collectivelyreferred to as the third light receiving elements 853. Also, theplurality of first light receiving elements 851, the plurality of secondlight receiving elements 852, and the plurality of third light receivingelements 853 may be collectively and simply referred to as the lightreceiving elements.

The first, second and third light receiving elements 851, 852 and 853receive light from a subject through the objective lens 112.Specifically speaking, the first light receiving elements 851 receivelight in a specified wavelength range and light in a first wavelengthrange different from the specified wavelength range. The specifiedwavelength range may be exemplified by the infrared range, such as thewavelength range of the luminescence light. The second light receivingelements 852 receive light in a second wavelength range different fromthe specified wavelength range. The third light receiving elements 853receive light in a third wavelength range different from the first,second and specified wavelength ranges.

The first, second and third wavelength ranges are different from eachother, and do not overlap each other. The first, second and third lightreceiving elements 851, 852 and 853 are arranged two-dimensionally in apredetermined pattern.

The spectral filter section 820 includes a plurality of filter elementseach of which transmits light in one of the first, second and thirdwavelength ranges. The filter elements are arranged two-dimensionally inaccordance with the first, second and third light receiving elements851, 852 and 853. Each light receiving element receives lighttransmitted by corresponding one filter element. In this manner, thefirst, second and third light receiving elements 851, 852 and 853receive light in different wavelength ranges.

The reception-side excitation light cut filter section 830 is providedat least between the subject and the second and third light receivingelements 852 and 853, and cuts off light in the excitation lightwavelength range. In this manner, the second and third light receivingelements 852 and 853 receive reflection light from the subject throughthe reception-side excitation light cut filter section 830. With such aconfiguration, the second and third light receiving elements 852 and 853substantially does not receive a portion of the excitation lightreflected by the subject.

The reception-side excitation light cut filter section 830 may cut offlight in the excitation light wavelength range and light in thespecified wavelength range. In this case, the second and third lightreceiving elements 852 and 853 receive substantially no luminescencelight from the subject, for example.

The reception-side excitation light cut filter section 830 may beprovided between the subject and the first light receiving elements 851.In this case, the reception-side excitation light cut filter section 830provided between the subject and the first light receiving elements 851is configured to transmit the light in the specified wavelength range.

The reception-side excitation light cut filter section 830 may includefilter elements arranged two-dimensionally in accordance with the first,second and third light receiving elements 851, 852 and 853, similarly tothe spectral filter section 820. In this case, the filter elementssupplying light to the first light receiving elements 851 at leasttransmit the light in the first and specified wavelength ranges. Thefilter elements supplying light to the first light receiving elements851 may cut off the light in the excitation light wavelength range. Thefilter elements supplying light to the second light receiving elements852 cut off the light in the excitation light and specified wavelengthranges and at least transmit the light in the second wavelength range.The filter elements supplying light to the third light receivingelements 853 cut off the light in the excitation light and specifiedwavelength ranges and at least transmit the light in the thirdwavelength range.

The image processing section 140 determines the pixel value of one pixelbased at least on the amount of light received by the first lightreceiving element 851 a, the second light receiving element 852 a, thesecond light receiving element 852 b and the third light receivingelement 853 a. In other words, a two-dimensional arrangement of thefirst light receiving element 851 a, the second light receiving element852 a, the second light receiving element 852 b and the third lightreceiving element 853 a forms a single pixel element. By arranging sucharrangements two-dimensionally, a plurality of pixel elements areformed. How to arrange the light receiving elements is not limited tothe configuration shown in FIG. 6 and may be modified in a variety ofmanners.

FIG. 7 illustrates exemplary spectral sensitivity characteristics of thefirst, second and third light receiving elements 851, 852 and 853. Lines930, 910, and 920 respectively show spectral sensitivity distributionsof the first, second and third light receiving elements 851, 852 and853. For example, the first light receiving elements 851 are sensitiveto light having a wavelength in the vicinity of 650 nm, where the otherlight receiving elements are substantially insensitive to 650 nm. Thesecond light receiving elements 852 are sensitive to light having awavelength in the vicinity of 450 nm, where the other light receivingelements are substantially insensitive to 450 nm. The third lightreceiving elements 853 are sensitive to light having a wavelength in thevicinity of 550 nm, where the other light receiving elements aresubstantially insensitive to 550 nm.

Also, the first light receiving elements 851 can receive light in theinfrared range (for example, 810 nm), which is an example of thespecified wavelength range. The above-described spectral sensitivitycharacteristics are dependent on the transmission characteristics of thereception-side excitation light cut filter section 830 and spectralfilter section 820 and the spectral sensitivities of the light receivingelements.

Having the above-described configurations, the first, second and thirdlight receiving elements 851, 852 and 853 respectively receiveR-component light, B-component light and G-component light. The firstlight receiving elements 851 can also receive light in the infraredrange, which is an example of the specified wavelength range. The first,second and third light receiving elements 851, 852 and 853 may be imagecapturing elements such as CCDs or CMOSs, for example. The first, secondand third light receiving elements 851, 852, and 853 respectively havespectral sensitivity characteristics shown by the lines 930, 910 and920, which are determined by the spectral transmittance of thereception-side excitation light cut filter section 830, the spectraltransmittances of the filter elements included in the spectral filtersection 820, and the spectral sensitivities of the image capturingelements.

FIG. 8 illustrates an exemplary configuration of the light irradiatingsection 150. The light irradiating section 150 includes a light emittingsection 1010 and a source-side filter section 1020. The light emittingsection 1010 emits light whose wavelength range covers the excitationlight, first, second and third wavelength ranges. In the presentembodiment, the light emitting section 1010 may be a xenon lamp, as anexample.

FIG. 9 illustrates an exemplary configuration of the source-side filtersection 1020. FIG. 9 illustrates the configuration of the portion of thesource-side filter section 1020 which faces the light emitting section1010. The source-side filter section 1020 includes an irradiation lightcut filter section 1120 and an excitation light cut filter section 1110.The light irradiation control section 170 rotates the source-side filtersection 1020 with respect to the central axis of the source-side filtersection 1020 within a plane substantially perpendicular to the directionin which the light emitted by the light emitting section 1010 travels.

The excitation light cut filter section 1110 transmits the light in thefirst wavelength range, the light in the second wavelength range, andthe light in the third wavelength range, and substantially cuts off thelight in the excitation light wavelength range. The irradiation lightcut filter section 1120 transmits the light in the excitation lightwavelength range, the light in the second wavelength range and the lightin the third wavelength range. It is preferable that the irradiationlight cut filter section 1120 substantially cuts off the light in thefirst wavelength range. The light from the light emitting section 1010is guided to a position off the central axis of the source-side filtersection 1020.

Therefore, when the light from the light emitting section 1010 is guidedto the excitation light cut filter section 1110, the light in theexcitation light wavelength range, out of the light from the lightemitting section 1010, is substantially cut off by the excitation lightcut filter section 1110, and the light in the first wavelength range,the light in the second wavelength range and the light in the thirdwavelength range are transmitted by the excitation light cut filtersection 1110. Therefore, at this timing, the light in the firstwavelength range, the light in the second wavelength range and the lightin the third wavelength range are substantially irradiated to thesubject.

On the other hand, when the light from the light emitting section 1010is guided to the irradiation light cut filter section 1120, the light inthe excitation light wavelength range, the light in the secondwavelength range and the light in the third wavelength range, out of thelight from the light emitting section 1010, are transmitted by theirradiation light cut filter section 1120. Therefore, at this timing,the excitation light, the light in the second wavelength range and thelight in the third wavelength range are substantially irradiated to thesubject.

The image capturing section 110 receives reflection light reflected fromthe specimen 20, when the light in the first wavelength range, the lightin the second wavelength range, and the light in the third wavelengthrange are irradiated to the specimen 20, under the control of the imagecapturing control section 160. Here, the light in the first wavelengthrange, the light in the second wavelength range, and the light in thethird wavelength range are visible light. Based on the amount of thelight received by the image capturing section 110, the surface imageobtaining section 214 generates a subject image by using the visiblelight, where the subject image is an example of the surface image. Whenthe irradiated light is substantially white light, the surface image maybe referred to as a white light image.

The image capturing section 110 receives the luminescence light emittedby the ICG in the subject, a portion of the light in the secondwavelength range which is reflected by the specimen 20, and a portion ofthe light in the third wavelength range which is reflected by thespecimen 20, when the excitation light, the light in the secondwavelength range, and the light in the third wavelength range areirradiated to the specimen 20, under the control of the image capturingcontrol section 160. The object image obtaining section 210 obtains asignal corresponding to the amount of the light received by the firstlight receiving elements 851 from the first light receiving elements851, and generates a subject image of luminescence light based on theamount of the luminescence light received by the first light receivingelements 851. The surface image obtaining section 214 generates asubject image of visible light based on the received amount of the lightin the second wavelength range corresponding to a signal from the secondlight receiving elements 852, the received amount of the light in thethird wavelength range corresponding to a signal from the third lightreceiving elements 853, and the amount of the light in the firstwavelength range received by the first light receiving elements 851 at adifferent timing.

FIG. 10 illustrates the image capturing timings of the image capturingsection 110 and exemplary images generated by the image processingsection 140. The image capturing control section 160 causes the imagecapturing section 110 to capture images by using light from an object attimes t1200, t1201, t1202, t1203, . . . . The light irradiation controlsection 170 causes the light emitted by the light emitting section 1010to be irradiated to the subject through the excitation light cut filtersection 1110 at a first timing exemplified by the times t 1200, t1201,and t1203, under the timing control of the image capturing controlsection 160. As stated here, under the control of the light irradiationcontrol section 170, the light irradiating section 150 irradiates thesubject with light whose wavelength range covers the first, second andthird wavelength ranges at the first timing.

At the first timing, the image capturing control section 160 controlsthe first light receiving elements 851 to receive the light in the firstwavelength range, controls the second light receiving elements 852 toreceive the light in the second wavelength range, and controls the thirdlight receiving elements 853 to receive the light in the thirdwavelength range, where the received light is included in the reflectionlight reflected by the subject when the subject is irradiated with thelight whose wavelength range covers the first, second and thirdwavelength ranges. In other words, at the first timing, the imagecapturing control section 160 controls the first light receivingelements 851 to receive the light in the first wavelength range from thesubject, controls the second light receiving elements 852 to receive thelight in the second wavelength range from the subject, and controls thethird light receiving elements 853 to receive the light in the thirdwavelength range from the subject.

On the other hand, at a second timing exemplified by the time t1202, thelight irradiation control section 170 controls the light emitted fromthe light emitting section 1010 to be irradiated to the subject throughthe irradiation light cut filter section 1120, under the timing controlof the image capturing control section 160. In other words, at thesecond timing, the light irradiating section 150 irradiates the subjectwith the light whose wavelength range covers the excitation lightwavelength range, the second wavelength range and the third wavelengthrange, under the control of the light irradiation control section 170.

At the second timing, the image capturing control section 160 controlsthe first light receiving elements 851 to receive the light in thespecified wavelength range emitted by the subject. That is to say, theimage capturing control section 160 controls the first light receivingelements 851 to receive the light in the specified wavelength range fromthe subject at the second timing.

As described above, at the second timing, the control section 105controls the light in the first wavelength range not to be irradiated tothe subject and controls the excitation light, the light in the secondwavelength range and the light in the third wavelength range to beirradiated to the subject, so that the first light receiving elements851 receive the light in the specified wavelength range emitted by thesubject, the second light receiving elements 852 receive the light inthe second wavelength range out of the reflection light from the subjectand the third light receiving elements 853 receive the light in thethird wavelength range out of the reflection light from the subject. Theexcitation light wavelength range is different from any of the first,second and third wavelength ranges, and does not overlap any of thefirst, second and third wavelength ranges.

As described above, the control section 105 controls the spectrum of thelight received by the first light receiving elements 851, the spectrumof the light received by the second light receiving elements 852 and thespectrum of the light received by the third light receiving elements853. The image processing section 140 generates an image by using lighthaving various wavelength ranges based on the amount of light receivedby the light receiving elements at each timing.

Specifically speaking, the surface image obtaining section 214 generatessubject images 1220 a, 1220 b, and 1220 d based on the amount of lightreceived by the light receiving elements at timings exemplified by thetimes t1200, t1201 and t1203. The subject images 1220 a, 1220 b, and1220 d can be substantially treated as visible light images obtainedwhen white light is irradiated. The subject image 1220 a includes ablood vessel image 1222 a and a blood vessel image 1224 a, the subjectimage 1220 b includes a blood vessel image 1222 b and a blood vesselimage 1224 b, and the subject image 1220 d includes a blood vessel image1222 d and a blood vessel image 1224 d.

In addition to the blood vessel images, the subject images 1220 a, 1220b, and 1220 d include surface images showing a surface of a physicalbody. As explained above, the surface image obtaining section 214generates a surface image of a subject at a first timing, by using thelight in the first wavelength range received by the first lightreceiving elements 851 at the first timing, the light in the secondwavelength range received by the second light receiving elements 852 atthe first timing, and the light in the third wavelength range receivedby the third light receiving elements 853 at the first timing.

The object image obtaining section 210 generates a subject image 1220 cincluding blood vessel images 1222 c, 1224 c, and 1226 c, based on theamount of light received by the light receiving elements at a timingexemplified by the time t1202. The subject image 1220 c can be treatedas a subject image formed by luminescence light from a subject. Thesubject image 1220 c is subjected to the above-described blur correctingoperation by the object image correcting section 220.

The subject image generating section 280 generates a subject image 1230c including blood vessel images 1232 c and 1234 c, based on the amountof the light received by the first light receiving elements 851 at atiming exemplified by the time t1201, the amount of light received bythe second light receiving elements 852 at a timing exemplified by thetime t1202 and the amount of the light received by the third lightreceiving elements 853 at the timing exemplified by the time t1202. Thesubject image 1230 c can be treated as a subject image of visible lightthat is supposed to be obtained at the timing exemplified by the timet1202.

As described above, the image processing section 140 generates a subjectimage of visible light for the second timing, by using the light in thefirst wavelength range received by the first light receiving elements851 at the first timing and the light in the second wavelength rangereceived by the second light receiving elements 852 at the secondtiming. Hence, the image processing section 140 can generate an image ofvisible light even at a timing where a luminescence light image iscaptured. The output section 180 successively displays the subjectimages 1220 a, 1220 b, 1230 c, 1220 d, . . . , thereby providing a videoimage with no frames dropped.

When the specimen 20 is a living organism having red bloods, forexample, a human body, a visible light image is usually characterized inthat the R component has a smaller spatial frequency component than theG and B components. For this reason, image degradation is usually lesssignificant when the R-component frame images are dropped than when theG- and B-component frame images are dropped. The above-describedconfiguration can thus reduce the awkwardness in the resulting videoimage, when compared with the case where the G- and B-component frameimages are dropped. As a result, the image processing system 10 may becapable of providing a visible light video image with substantially noframe images dropped.

As discussed above, the image processing system 10 can capture thesubject image 1220 c by using the luminescence light in the infraredwavelength range generated from the specimen 20 by the excitation lightin the infrared region. Having a longer wavelength than visible light,the excitation light is more difficult to be absorbed by a physicalmatter than visible light. Therefore, the excitation light can reach alarger depth (for example, approximately 1 cm) in a physical matter thanvisible light and causes the specimen 20 to emit luminescence light. Theluminescence light has a further longer wavelength than the excitationlight and thus can easily reach the surface of the physical matter. As aconsequence, the image processing system 10 can provide the subjectimage 1220 c including a blood vessel image 1226 d showing a bloodvessel in a very deep region, which is not included in the subjectimages 1220 a, 1220 b and 1220 d obtained by visible light.

The output section 180 may generate a combined image by combining thesubject image 1220 c and one of the subject images 1220 b and 1220 dcaptured at timings in the vicinity of the timing at which the subjectimage 1220 c is captured and output the combined image to outside. Forexample, the output section 180 may display the combined image.Alternatively, the output section 180 may record the subject image 1220c in association with one of the subject images 1220 b and 1220 d.

At a timing where a visible light image is captured, the control section105 cuts off light in the excitation light wavelength range and light inthe luminescence light wavelength range in the light from the lightemitting section 1010, and irradiates the subject with the resultantlight. Therefore, the image processing system 10 can provide a visiblelight image showing a surface of a physical matter, which shows no bloodvessels inside the physical matter and thus is suitable for the physicalmatter surface observation.

FIG. 11 illustrates how to generate a motion-compensated surface image.FIG. 10 is used to describe an exemplary operation to generate thesubject image 1230 c by multiplexing the R signal corresponding to theamount of the light received by the first light receiving elements 851at the time t1201 with the B and G signals corresponding to the amountof the light received by the second light receiving elements 852 at thetime t1202 and the amount of the light received by the third lightreceiving elements 853 at the time t1202, under an assumption that thereare no factors to trigger a significant change in images over time, suchas motion of the end portion 102 of the endoscope 100 and motion of thespecimen 20 for the sake of intelligibility. Referring to the operation,in fact, motion of the end portion 102 of the endoscope 100, motion ofthe specimen 20 and the like may cause a disagreement between the Rsignal and the other color signals in the visible light image.

With reference to FIGS. 11 and 12, the following describes theoperations and functions of the image processing section 140 to correctthe influence on the visible light image by various factors includingthe above-mentioned motions, especially with a main focus on theoperations of the motion identifying section 270 and the subject imagegenerating section 280.

The motion identifying section 270 uses images formed by the B signal ata plurality of timings to identify a motion of an object among thoseimages. Here, the motion of the object is a motion that causes a changein the images over time, and is exemplified by the motion of thespecimen 20, the motion of the end portion 102 of the endoscope 100, andthe change over time in the zoom value of the image capturing section110. The motion of the end portion 102 of the endoscope 100 includes achange over time in the position of the end portion 102 which causes achange over time in the image capturing position of the image capturingsection 110, and a change over time in the facing direction of the endportion 102 which causes a change over time in the image capturingdirection of the image capturing section 110.

The motion identifying section 270 identifies the motion of the objectbased on the B-signal images at the times t1201 and t1202. For example,the motion identifying section 270 may identify the motion of the objectby attempting to match objects extracted from a plurality of images toeach other.

The subject image generating section 280 corrects an R signal obtainedat the time t1201 according to the identified motion, to generate an Rsignal that is supposed to be obtained at the time t1202. The subjectimage generating section 280 then multiplexes together the R signalgenerated by the compensation process, the B signal obtained at the timet1202, and the G signal obtained at the time t1202, to generate asurface image for the time t1202.

An image 1321 b is formed by the R signal from the first light receivingelements 851 at the time t1201. Images 1322 b and 1322 c arerespectively formed by the B signal from the second light receivingelements 852 at the times t1201 and t1202. Images 1323 b and 1323 c arerespectively formed by the G signal from the third light receivingelements 853 at the times t1201 and t1202.

In the present example, the motion identifying section 270 identifiesthe motion by referring to what is shown in the images 1322 b and 1322c. Specifically speaking, the motion identifying section 270 extractsobjects showing the same subject from the images 1322 b and 1322 c.According to the example shown in FIG. 11, the motion identifyingsection 270 extracts objects 1352 b and 1352 c from the images 1322 band 1322 c.

The motion identifying section 270 calculates the difference in positionbetween the objects 1352 b and 1352 c. According to the example shown inFIG. 11, it is assumed that the difference in position is seen in a ydirection on the images for the intelligibility. In such a case, themotion identifying section 270 calculates the positional difference Δy1between the objects 1352 b and 1352 c.

The subject image generating section 280 shifts the image 1321 b in they direction by an amount determined in accordance with the calculatedpositional difference Δy1, thereby generating an image 1321 c. Thesubject image generating section 280 combines the images 1321 c, 1322 c,and 1323 c, to generate a surface image 1330 c. Here, the combiningprocess includes multiplexing the R signal representing the image 1321c, the B signal representing the image 1322 c and the G signalrepresenting the image 1323 c with predetermined weights.

According to the above description, the motion is identified by usingthe images 1322 formed by the B signal. In a similar manner, however,the motion can be identified by using the images 1323 formed by the Gsignal. The motion identifying section 270 may identify a motion byusing images in one of the wavelength ranges which is selected based onthe contrast levels of the captured images. For example, the motionidentifying section 270 may identify a motion by using the highestcontrast images with the highest priority. The motion identifyingsection 270 may be capable of more accurately identifying a motion byusing B-signal images when microstructure images can be used for motionidentification, which may become possible because clear images areavailable for microstructures on a surface. Furthermore, the motionidentifying section 270 may be capable of more accurately identifying amotion by using G-signal images when concave and convex structure imagescan be used for motion identification, which may become possible becauseclear images are available for concave and convex structures on asurface.

The subject image generating section 280 may compensate the motion in anR-signal image by a different amount in each image region. For example,when the image capturing direction of the image capturing section 110 isperpendicular to the surface of the subject and the end portion 102 ofthe endoscope 100 moves horizontally with respect to the surface of thesubject, the amount of the motion of an object can be assumed to be thesame in every image region. On the other hand, when the image capturingdirection of the image capturing section 110 is not perpendicular to thesurface of the subject, for example, the amount of the motion may besmaller in an image region showing a distant region from the end portion102 than in an image region showing a close region to the end portion102.

In order that the subject image generating section 280 calculates thedegree of the motion compensation on an R-signal image for each imageregion, the positional relation between the surface of the subject andthe image capturing section 110 may need to be known or estimated. Basedon the positional relation and the position of each image region, thesubject image generating section 280 can calculate the degree of themotion compensation for each image region. The subject image generatingsection 280 may obtain control values used for operating the endoscope100, which may influence changes over time in images, such as controlvalues used for controlling the position and facing direction of the endportion 102 and control values used to control the zoom value of theimage capturing section 110 and calculate the degree of the motioncompensation on an R-signal image with reference to the obtained controlvalues.

Alternatively, the motion identifying section 270 may calculate a motionof an object in each image region. The subject image generating section280 may calculate the degree of the motion compensation on each imageregion based on the identified motion of the object in each imageregion.

When identifying a motion in each image region, the motion identifyingsection 270 may identify the motion in each image region by using imagesformed by light in one of the wavelength ranges which is determined inassociation with each image region. For example, the motion identifyingsection 270 calculates the contrast of each image in units of imageregions. The motion identifying section 270 may select, in associationwith each image region, images formed by light in a wavelength range forwhich the highest contrast is calculated in the highest prioritycompared with images formed by light in the other wavelength ranges, andidentify a motion of an object by using the selected images.

As described above with reference to FIGS. 10 and 11, the motionidentifying section 270 uses an image formed by the light in the secondwavelength range received by the second light receiving elements 852 ata first timing and an image formed by the light in the second wavelengthrange received by the second light receiving elements 852 at a secondtiming in order to identify a motion of an object on the images betweenthe first and second timings. The subject image generating section 280generates a surface image of the second timing, by using the light inthe first wavelength range received by the first light receivingelements 851 at the first timing, the light in the second wavelengthrange received by the second light receiving elements 852 at the secondtiming and the identified motion.

FIG. 12 illustrates another exemplary method to generate amotion-compensated surface image. According to the example shown in FIG.12, the motion identifying section 270 identifies a motion of an objectby using an R-signal image 1421 a obtained at the time t1200 and anR-signal image 1421 b obtained at the time t1201. Similarly to themethod described with reference to FIG. 11, the motion identifyingsection 270 extracts objects showing the same subject from the images1421 a and 1421 b. According to the example shown in FIG. 12, the motionidentifying section 270 extracts objects 1451 a and 1451 b respectivelyfrom the images 1421 a and 1421 b.

The motion identifying section 270 then calculates the difference inposition between the objects 1451 a and 1451 b. According to the exampleshown in FIG. 12, it is also assumed that the difference in position isseen in a y direction on the images for the intelligibility. In such acase, the motion identifying section 270 calculates the positionaldifference Δy2 between the objects 1451 a and 1451 b. In the same manneras described with reference to FIG. 11, the subject image generatingsection 280 generates an image 1421 c by shifting the image 1421 b inthe y direction by an amount determined in accordance with thecalculated positional difference Δy2. The subject image generatingsection 280 combines the image 1421 c, an image 1422 c formed by the Bsignal from the second light receiving elements 852 at the time t1202,and an image 1423 c formed by the G signal from the third lightreceiving elements 853 at the time t1202, thereby generating a surfaceimage 1430 c.

According to the above description, the images 1421 a and 1421 b areused to identify the motion. The motion identifying section 270,however, may identify the motion by using the image 1421 b and anR-signal image obtained at a time t1203. In other words, the motionidentifying section 270 may identify a motion by using images obtainedat a plurality of timings including timings preceding and following thetime t1202 for which a motion-compensated R-signal image is generated.When it is permitted to display a visible light image with a delay to acertain degree, the motion identifying section 270 may be capable ofidentifying a motion more accurately by using an image captured at alater timing.

As described above with reference to FIG. 12, the motion identifyingsection 270 uses a plurality of images formed by the light in the firstwavelength range received by the first light receiving elements 851 at aplurality of timings including not a second timing but a first timing,in order to identify a motion of an object on the images between thetimings. The subject image generating section 280 generates a surfaceimage for the second timing, with reference to the light in the firstwavelength range received by the first light receiving elements 851 atthe first timing, the light in the second wavelength range received bythe second light receiving elements 852 at the second timing, and theidentified motion.

According to the exemplary motion identifying operations described withreference to FIGS. 11 and 12, the motion identifying section 270identifies a motion by using images captured at two timings. The motionidentifying section 270, however, may identify a motion by using imagescaptured at three or more timings. Furthermore, the motion identifyingsection 270 can select, for each image region, images of a particularwavelength range in order to identify a motion, from R-signal images inaddition to B-signal and G-signal images.

The object image correcting section 220 can use the motion identified bythe motion identifying section 270 in order to identify which bloodvessel images 1222 and 1224 in different subject images 1220 correspondto the blood vessel images 1222 c and 1224 c included in the subjectimage 1220 c, when performing the above-described blur correction on thesubject image 1220 c.

The second and third light receiving elements 852 and 853 are sensitiveto light in the luminescence light wavelength range, and may receiveluminescence light from a subject at a timing exemplified by the timet1202. In this case, the spectral filter section 820 and thereception-side excitation light cut filter section 830 may transmit thelight in the luminescence light wavelength range.

In this case, the object image obtaining section 210 may generate anobject image by performing pixel addition processing. Specificallyspeaking, the object image obtaining section 210 adds together imagesignals from a plurality of light receiving elements in the vicinity ofeach other, which are selected from the first, second and third lightreceiving elements 851, 852 and 853. Here, an image signal from a lightreceiving element may be a signal indicating a charge amount determinedin accordance with the amount of light received by the light receivingelement. This signal representing the charge amount may be an analogsignal determined by the amount of light received by the light receivingelement, or a digital signal obtained by A/D converting the analogsignal. Any pixel addition processing can increase a signal component.Here, an increase in a random noise component caused by the pixeladdition processing is smaller than the increase in the signal componentcaused by the pixel addition processing. Therefore, the aboveconfiguration can improve the S/N ratio when compared with a case wherepixel addition processing is not applied.

In the same manner as described with reference to FIGS. 11 and 12, themotion identifying section 270 can identify a motion by using theR-signal, G-signal or B-signal images obtained at a plurality of timingsexcluding a timing exemplified by the time t1202. The subject imagegenerating section 280 can generate a visible light subject image whichis supposed to be obtained at the timing exemplified by the time t1202by correcting, according to the identified motion, a visible lightsubject image obtained at a timing excluding the timing exemplified bythe time t1202.

According to the above description of the exemplary configuration of thelight irradiating section 150, a single light source and a rotationfilter are used, where the light source can emit light whose wavelengthrange includes the visible light wavelength range and the excitationlight wavelength range. As an alternative example, the light irradiatingsection 150 can emit visible light and light containing excitation lightin a time-sharing manner, by controlling light emission of a pluralityof light emitting elements each of which is designed to emit light inone of a plurality of different wavelength ranges. For example, a lightemitting element designed to emit visible light can be exemplified by asemiconductor element such as an LED, and a light emitting elementdesigned to emit excitation light can be exemplified by a semiconductorelement such as semiconductor laser. Alternatively, a light emittingelement can be formed by using a fluorescence substance that emitsluminescence light such as fluorescence when excited.

The light irradiation control section 170 can make it possible to emitvisible light and light containing excitation light in a time-sharingmanner, by controlling the light emission intensity of each lightemitting element at each timing. Here, “controlling the light emissionintensity of each light emitting element at each timing” includescontrolling a different combination of light emitting elements to emitlight at each timing.

The light emitting elements may be provided at the end portion 102 ofthe endoscope 100. The light emitting elements may emit light whenelectrically or optically excited. When the light emitting elements emitlight when optically excited, the light irradiating section 150 mayinclude the light emitting elements and an exciting section that emitslight for exciting the light emitting elements. The light emittingelements may emit light having a different spectrum according to thewavelength of the excitation light. In this case, the light irradiationcontrol section 170 can control the spectrum of irradiation light bycontrolling the wavelength of the excitation light emitted by theexciting section at each timing. When the plurality of light emittingelements are excited by the same excitation light, each light emittingelement may emit light with a different spectrum. Here, a portion of theexcitation light which passes through the light emitting elements may beirradiated to a subject as irradiation light.

According to the above description of the exemplary configuration of theimage capturing section 110, the spectral filter section 820 is providedat the light-reception side. As an alternative configuration, the imagecapturing section 110 may not include the spectral filter section 820.In this case, the light irradiating section 150 may provide light in theR wavelength range, light in the G wavelength range, light in the Bwavelength range, and light in the excitation light wavelength range ina time-sharing manner. The surface image obtaining section 214 cangenerate a visible light subject image by multiplexing together signalsfrom a plurality of light receiving elements at timings where visiblelight is irradiated. The object image obtaining section 210 can generatea luminescence light subject image by using a signal from a lightreceiving element at a timing where excitation light is irradiated.

To provide light in the R wavelength range, light in the G wavelengthrange, light in the B wavelength range, and light in the excitationlight wavelength range in a time-sharing manner, the light irradiatingsection 150 may be configured by including one or more light sourcesthat can emit light whose wavelength range includes the above-mentionedvisible light and excitation light wavelength ranges and a rotationfilter that includes a plurality of filter sections each mainly andselectively transmitting light in a corresponding one of the wavelengthranges. Alternatively, the light irradiating section 150 may beconfigured so as to control light emission of a plurality of lightemitting elements each of which is designed to emit light in a differentwavelength range, as described above.

Even when light in each wavelength range is irradiated in a time-sharingmanner, the motion identifying section 270 can identify a motion byusing image signals of one color component obtained at a plurality oftimings, in the same manner as described with reference to FIGS. 11 and12. The subject image generating section 280 may use, for example, anR-signal image obtained at a timing where the light in the R wavelengthrange is irradiated and the identified motion in order to generate anR-signal image which is supposed to be obtained at a different timing atwhich the light in the R wavelength range is not irradiated. In asimilar manner, the subject image generating section 280 generates aG-signal image for a timing at which the light in the G wavelength rangeis not irradiated and a B-signal image for a timing at which the lightin the B wavelength range is not irradiated. In this way, the subjectimage generating section 280 can generate a visible light surface imagewhich is supposed to be obtained at each timing.

FIG. 13 illustrates an exemplary hardware configuration of the imageprocessing system 10 relating to the embodiment of the presentinvention. The image processing system 10 relating to the presentembodiment is constituted by a CPU peripheral section, an input/output(I/O) section and a legacy I/O section. The CPU peripheral sectionincludes a CPU 1505, a RAM 1520, a graphic controller 1575 and a displaydevice 1580 which are connected to each other by means of a hostcontroller 1582. The I/O section includes a communication interface1530, a hard disk drive 1540, and a CD-ROM drive 1560 which areconnected to the host controller 1582 by means of an I/O controller1584. The legacy I/O section includes a ROM 1510, a flexible disk drive1550, and an I/O chip 1570 which are connected to the I/O controller1584.

The host controller 1582 connects the RAM 1520 with the CPU 1505 andgraphic controller 1575 which access the RAM 1520 at a high transferrate. The CPU 1505 operates in accordance with programs stored on theROM 1510 and RAM 1520, to control the constituents. The graphiccontroller 1575 obtains image data which is generated by the CPU 1505 orthe like on a frame buffer provided within the RAM 1520, and causes thedisplay device 1580 to display the obtained image data. Alternatively,the graphic controller 1575 may include therein a frame buffer forstoring thereon the image data generated by the CPU 1505 or the like.

The I/O controller 1584 connects, to the host controller 1582, the harddisk drive 1540, communication interface 1530 and CD-ROM drive 1560which are I/O devices operating at a relatively high rate. Thecommunication interface 1530 communicates with different apparatuses viathe network. The hard disk drive 1540 stores thereon programs and datato be used by the CPU 1505 in the image processing system 10. The CD-ROMdrive 1560 reads programs or data from a CD-ROM 1595, and supplies theread programs or data to the hard disk drive 1540 via the RAM 1520.

The I/O controller 1584 is also connected to the ROM 1510, flexible diskdrive 1550 and I/O chip 1570 which are I/O devices operating at arelatively low rate. The ROM 1510 stores thereon a boot program executedby the image processing system 10 at the startup, programs dependent onthe hardware of the image processing system 10, and the like. Theflexible disk drive 1550 reads programs or data from a flexible disk1590, and supplies the read programs or data to the hard disk drive 1540via the RAM 1520. The I/O chip 1570 is connected to the flexible diskdrive 1550, and used to connect a variety of I/O devices to the imageprocessing system 10, via a parallel port, a serial port, a keyboardport, a mouse port or the like.

The communication programs to be provided to the hard disk drive 1540via the RAM 1520 are provided by a user in the state of being stored ona recording medium such as the flexible disk 1590, the CD-ROM 1595, andan IC card. The communication programs are read from the recordingmedium, and the read programs are installed in the hard disk drive 1540in the image processing system 10 via the RAM 1520, to be executed bythe CPU 1505. The communication programs that are installed in the imageprocessing system 10 and executed cause the CPU 1505 and the like tocause the image processing system 10 to function as the respectiveconstituents included in the image processing system 10 described withreference to FIGS. 1 to 12. For example, the programs cause the imageprocessing system 10 to function as the image capturing section 110, theimage processing section 140, the output section 180, the lightirradiating section 150, the control section 105 and the like describedwith reference to FIGS. 1 to 12.

Although some aspects of the present invention have been described byway of exemplary embodiments, it should be understood that those skilledin the art might make many changes and substitutions without departingfrom the spirit and the scope of the present invention which is definedonly by the appended claims.

1. An image processing system comprising: an object image obtainingsection that obtains an object image formed by light from an objectinside a physical body; a depth identifying section that identifies adepth from a surface of the physical body to the object; a distanceinformation identifying section that identifies distance informationindicating a distance from an image capturing section capturing theobject image to the surface of the physical body; and an object imagecorrecting section that corrects the object image according to thedistance information and the depth.
 2. The image processing system asset forth in claim 1, wherein the object image correcting sectioncorrects spread of the object image according to the distanceinformation and the depth.
 3. The image processing system as set forthin claim 2, wherein the object image correcting section corrects thespread of the object image that occurs because the light from the objectis scattered between the object and the surface, according to thedistance information and the depth.
 4. The image processing system asset forth in claim 2, further comprising a correction table that storesa correction value used to correct the spread of the object image, inassociation with the depth from the surface to the object, wherein theobject image correcting section corrects the spread of the object imagewith reference to the distance information, the depth identified by thedepth identifying section and the correction value.
 5. The imageprocessing system as set forth in claim 4, wherein the correction tablestores the correction value in a real space in association with thedepth from the surface to the object, and the object image correctingsection includes: a correction value transforming section thattransforms the correction value in a real space into a correction valuein the object image, according to the distance information; and anobject image correction section that corrects the spread of the objectimage by using the correction value in the object image produced by thecorrection value transforming section.
 6. The image processing system asset forth in claim 2, wherein the object image correcting sectionincreases the correction on the spread of the object image as the depthincreases.
 7. The image processing system as set forth in claim 2,wherein the object image correcting section increases the correction onthe spread of the object image as the distance indicated by the distanceinformation decreases.
 8. The image processing system as set forth inclaim 2, further comprising a light image obtaining section that obtainsa plurality of light images each of which is captured by using light,from the object, in one of a plurality of different wavelength ranges,wherein the depth identifying section identifies the depth withreference to what is shown in the plurality of light images.
 9. Theimage processing system as set forth in claim 8, further comprising anobject region identifying section that identifies an image region of theobject in each of the plurality of light images, wherein the depthidentifying section identifies the depth with reference to luminance inthe image region identified by the object region identifying section.10. The image processing system as set forth in claim 8, wherein thelight image obtaining section obtains the plurality of light images byusing light rays belonging to a plurality of different wavelength rangesincluding a light ray emitted from a luminescence substance inside theobject.
 11. The image processing system as set forth in claim 10,further comprising a display control section that controls how todisplay the object image corrected by the object image correctingsection, according to the depth.
 12. The image processing system as setforth in claim 11, wherein the display control section changesbrightness or a color of the object image corrected by the object imagecorrecting section, according to the depth.
 13. The image processingsystem as set forth in claim 8, wherein the light image obtainingsection obtains the plurality of light images by using light reflectedby the object.
 14. The image processing system as set forth in claim 13,wherein the light image obtaining section obtains the plurality of lightimages by using light rays belonging to a plurality of differentwavelength ranges included in light reflected from the object when theobject is irradiated with white light.
 15. The image processing systemas set forth in claim 13, wherein the light image obtaining sectionobtains the plurality of light images by using light rays reflected fromthe object when the object is irradiated with light rays belonging to aplurality of different wavelength ranges.
 16. The image processingsystem as set forth in claim 1, further comprising a surface imageobtaining section that obtains an image including a surface image thatshows the surface of the physical body and is formed by light irradiatedto the surface of the physical body in a direction substantiallyparallel to an image capturing direction of the image capturing section,wherein the distance information identifying section identifies thedistance information with reference to a size of the surface imageincluded in the image obtained by the surface image obtaining section.17. An image processing method comprising: obtaining an object imageformed by light from an object inside a physical body; identifying adepth from a surface of the physical body to the object; identifyingdistance information indicating a distance from an image capturingsection capturing the object image to the surface of the physical body;and correcting the object image according to the distance informationand the depth.
 18. A non-transitory computer readable medium including aprogram for use with an image processing system, execution of theprogram causing the image processing system to function as: an objectimage obtaining section that obtains an object image formed by lightfrom an object inside a physical body; a depth identifying section thatidentifies a depth from a surface of the physical body to the object; adistance information identifying section that identifies distanceinformation indicating a distance from an image capturing sectioncapturing the object image to the surface of the physical body; and animage correcting section that corrects the object image according to thedistance information and the depth.